Displaying estimated social interest in time-based media

ABSTRACT

Social media content items are mapped to relevant time-based media events. These mappings may be used as the basis for multiple applications, such as ranking of search results for time-based media, automatic recommendations for time-based media, prediction of audience interest for media purchasing/planning, and estimating social interest in the time-based media. Social interest in time-based media (e.g., video and audio streams and recordings) segments is estimated through a process of data ingestion and integration. The estimation process determines social interest in specific events represented as segments in time-based media, such as particular plays in a sporting event, scenes in a television show, or advertisements in an advertising block. The resulting estimates of social interest also can be graphically displayed.

PRIORITY INFORMATION

This application is a division of U.S. patent application Ser. No.12/838,402, filed on Jul. 16, 2010, entitled “Displaying EstimatedSocial Interest in Time-Based Media,” which application claims priorityunder 35 U.S.C. §119(e) to U.S. Provisional Patent Application No.61/226,002, filed on Jul. 16, 2009 and entitled “Method of EstimatingSocial Interest in American Football,” which are incorporated herein byreference in their entirety.

BACKGROUND OF THE INVENTION

The present invention relates generally to using social media toestimate interest in media events, and in particular to aggregatingsocial media content items and references to the media events thereinfor estimating social interest in time-based media.

Online social media services, such as social networking sites, searchengines, news aggregators, blogs, and the like provide a richenvironment for users to comment on events of interest and communicatewith other users. Content items contributed by users of these socialmedia services often include references to events that appear in timebased media such as television shows, news reports, sporting events,movies, concert performances, and the like. However, although thecontent items refer to the time-based media, the social media contentitems themselves typically are isolated from the events and time-basedmedia in which those events occur.

SUMMARY OF THE INVENTION

Social media content items and references to events that occur thereinare aligned with the time-based media events they describe. Thesemappings may be used as the basis for multiple applications, such asranking of search results for time-based media, automaticrecommendations for time-based media, prediction of audience interestfor media purchasing/planning, and estimating social interest in thetime-based media. Social interest in time-based media (e.g., video andaudio streams and recordings) segments is estimated through a process ofdata ingestion and integration. The estimation process determines socialinterest in specific segments of time-based media, such as particularplays in a sporting event, scenes in a television show, or steps in aninstructional video. The social interest in a given event is determinedby aggregating social media content items with confidence scoresindicating the likelihood that the content items refer to the givenevent.

For an event appearing in time-based media, which event may have beenidentified by segmentation of the time-based media, social media contentitems are identified as potentially relevant to the event. Theprobability that the content item is relevant to the time-based mediaevent is determined for each social media content item, and a confidencescore reflecting the probability is assigned to the content item.Content items with higher probabilities are aligned with the event,aggregated, and stored. The aggregated content items are associated withan aggregate score for the time-based media event, where the aggregatescore is an estimate of the level of social interest in the time-basedmedia event. The estimated level of social interest also can begraphically displayed. The features and advantages described in thissummary and the following detailed description are not all-inclusive.Many additional features and advantages will be apparent to one ofordinary skill in the art in view of the drawings, specification, andclaims hereof.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates the computing environment of one embodiment of asystem for associating social media content items with time-based mediaevents and determining social interest in the events based on theresulting associations.

FIG. 2 is a block diagram of one embodiment of a social interestinformation provider.

FIG. 3 is a conceptual diagram illustrating the video/metadataalignment/annotation and social media/event alignment processes at ahigh level according to one embodiment.

FIG. 3A is a flow diagram illustrating one embodiment of a method forassociating social media content items with time-based media events, anda related method of determining social interest in the events based onthe resulting associations.

FIG. 4 is a flow diagram illustrating one embodiment of a video eventsegmentation process.

FIG. 5 is a flow diagram illustrating one embodiment of a metadataalignment/annotation process.

FIG. 6 is a flow diagram illustrating one embodiment of a socialmedia/event alignment process.

FIG. 7 is a flow diagram illustrating one embodiment of a socialinterest estimation process.

FIGS. 8A and 8B show two embodiments of social interest heat mapsshowing levels of social interest for a plurality of eventscorresponding to a series of chronological time segments in a time-basedmedium.

FIGS. 9A-9C show three embodiments of user interfaces of a socialinterest estimation system.

FIGS. 10A and 10B show two embodiments of user interfaces of a socialinterest estimation system showing a sentiment view.

FIGS. 11A-11C show three embodiments of user interfaces of a socialinterest estimation system showing a filtered view.

FIG. 12A shows one embodiment of user interface of a social interestestimation system showing a focused unexpanded view.

FIG. 12B shows one embodiment of user interface of a social interestestimation system showing a focused expanded view.

FIGS. 13A-D show yet another embodiment of a user interface displayingsocial interest heat maps showing levels of social interest for aplurality of events corresponding to a series of chronological timesegments in a time-based medium.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION

FIG. 1 illustrates the computing environment 100 for one embodiment of asystem 130 for associating social media content items and references toevents therein with time-based media events and determining socialinterest in the events based on the resulting associations.

The environment 100 includes social media sources 110, time-based mediasources 120, the social interest information provider 130, a network140, and client devices 150.

The social media sources 110 include social networks, blogs, news media,forums, user groups, etc. These sources generally provide a plurality ofusers with the ability to communicate and interact with other users ofthe source. Users can typically contribute various content items (e.g.,posts, videos, photos, links, status updates, blog entries, tweets, andthe like), which may refer to media events, and can engage indiscussions, games, online events, and other participatory services.

The time-based media sources 120 include broadcasters, direct contentproviders, advertisers, and any other third-party providers oftime-based media content. These sources 120 typically publish contentsuch as television programs, videos, movies, serials, audio recordings,and the like.

The social interest information provider 130 provides a system forassociating social media content items and references to events thereinwith time-based media events and determining social interest in theevents based on the resulting associations, and is further described inconjunction with FIG. 2.

The network 140 may comprise any combination of local area and/or widearea networks, the Internet, or one or more intranets, using both wiredand wireless communication systems.

The client devices 150 comprise computing devices that can receive inputfrom a user and can transmit and receive data via the network 140. Forexample, client devices 150 may be a desktop computer, a laptopcomputer, a smart phone, a personal digital assistant (PDAs), or anyother device including computing functionality and data communicationcapabilities. A client device 150 is configured to communicate with thesocial media sources 110 and the social interest information providersystem 130 via the network 140.

FIG. 2 is a block diagram of one embodiment of a social interestinformation provider 130. The embodiment of the social interestinformation provider 130 shown in FIG. 2 is a computer system thatincludes a web server 200 and associated API 202, a domain ontologyengine 205, an author identifier 210, a closed captioning extractor 215,an event segmentation engine 220, a feature extraction engine 225, ametadata alignment engine 230, an annotation engine 235, a comparativefeature extraction engine 240, a media event/alignment engine 245, asocial interest estimator 250, a user interface engine 255, domainontologies 257, a social media content store 260, a social media authorstore 263, a usage stats store 265, a closed captioning store 267, amultimedia store 270, an event metadata store 273, a mapping store 275,a video event store 280, a social interest store 285, and an annotatedevent store 290. This system may be implemented using a single computer,or a network of computers, including cloud-based computerimplementations. The computers are preferably server class computersincluding one or more high-performance CPUs, 1G or more of main memory,as well as 500 GB to 2 Tb of computer readable, persistent storage, andrunning an operating system such as LINUX or variants thereof. Theoperations of the system 130 as described can be controlled througheither hardware or through computer programs installed in computerstorage and executed by the processors of such servers to perform thefunctions described herein. The system 130 includes other hardwareelements necessary for the operations described here, including networkinterfaces and protocols, security systems, input devices for dataentry, and output devices for display, printing, or other presentationsof data; these and other conventional components are not shown so as tonot obscure the relevant details.

As noted above, system 130 comprises a number of “engines,” which refersto computational logic for providing the specified functionality. Anengine can be implemented in hardware, firmware, and/or software. Anengine may sometimes be equivalently referred to as a “module” or a“server.” It will be understood that the named components represent oneembodiment of the present invention, and other embodiments may includeother components. In addition, other embodiments may lack the componentsdescribed herein and/or distribute the described functionality among thecomponents in a different manner. Additionally, the functionalitiesattributed to more than one component can be incorporated into a singlecomponent. Where the engines described herein are implemented assoftware, the engine can be implemented as a standalone program, but canalso be implemented through other means, for example as part of a largerprogram, as a plurality of separate programs, or as one or morestatically or dynamically linked libraries. In any of these softwareimplementations, the engines are stored on the computer readablepersistent storage devices of the system 130, loaded into memory, andexecuted by the one or more processors of the system's computers. Theoperations of the system 130 and its various components will be furtherdescribed below with respect to FIG. 2 and the remaining figures. Aswill become apparent, the various data processing operations describedherein are sufficiently complex and time consuming as to require theoperation of a computer system such as the system 130.

The web server 200 links the social interest information provider 130 tothe client devices 150, the time-based media sources 120, and the socialmedia sources 110 via network 140, and is one means for doing so. Theweb server 200 serves web pages, as well as other web related content,such as Java, Flash, XML, and so forth. The web server 200 may include amail server or other messaging functionality for receiving and routingmessages between the social interest information provider 130 and clientdevices 150.

The API 202, in conjunction with web server 200, allows one or moreexternal entities to access information from the social interestinformation provider 130. The web server 200 may also allow externalentities to send information to the social interest information provider130 calling the API 202. For example, an external entity sends an APIrequest to the social interest information provider 130 via the network140 and the web server 200 receives the API request. The web server 200processes the request by calling an API 202 associated with the APIrequest to generate an appropriate response, which the web server 200communicates to the external entity via the network 140. The API 202 canbe used for the social interest information provider 130 to receiveextracted features and other inputs to the social media/event alignment330 and social interest estimation 340 processes from third parties(such as entities providing the time based media), which then would beused by the social interest information provider 130 in those processes.

The action logger 215 is capable of receiving communications from theweb server 210 about user actions on and/or off the social networkingsystem 130. The action logger 215 populates the action log 230 withinformation about user actions, allowing the social networking system130 to track various actions taken by its users within the socialnetworking system 130 and outside of the social networking system 130.Any action that a particular user takes with respect to another user isassociated with each user's profile, through information maintained inthe action log 230 or in a similar database or other data repository.Examples of actions taken by a user within the social network 130 thatare identified and stored may include, for example, adding a connectionto another user, sending a message to another user, reading a messagefrom

Domain ontology engine 205 provides domain ontologies indicatingvocabularies specific to different media domains for storage in thedomain ontologies 257, and is one means for doing so. The domainontologies 257 encode information relevant to specific domains, and arebeneficial, since nicknames, slang, acronyms, and other shortened termscommonly are used in certain domains. Domain ontologies 257 may beorganized hierarchically as graphs, where each node in the graphrepresents a concept (e.g. “football play,” “scoring play”) and eachedge represents a relation between concepts (e.g. “type of”). Conceptinstances (e.g., a specific touchdown play from a specific footballgame) may also be encoded in the domain ontology, as well as,vocabularies that provide alternate terminology for concept nodes (e.g.“TD” for concept “touchdown”). The domain ontologies 257 may beengineered based on the knowledge of human experts or machine-generated.The domain ontologies are used for initial filtering of social mediaposts and in the social media/event alignment process. An exemplary listof social interest domains for which time-based media is used accordingto the present invention includes broadcast video such as televisionprograms, such as sports, news, episodic television, reality/live eventshows, movies, and advertising in conjunction with any of these domains.More specific domains also are possible, e.g., football games,entertainment news, specific reality TV shows, etc., each of which mayhave their own domain-specific ontology. The domain ontology engine 205is configured to filter the time segments according to a search term,wherein the graphical display displays only a subset of the series ofchronological time segments corresponding to the search term.

The author identifier 210 identifies the author, or provider, of eachsocial media content item, e.g., as provided to the social interestinformation provider 130 by the social media sources 110 with thecontent items, and is one means for doing so. Additional informationabout the authors may be extracted from the content items themselves,e.g., as stored in the social media content store 260, or extracted fromother external sources. The author information is stored in the socialmedia author store 263.

The closed captioning extractor 215 extracts closed captioning data fromthe time-based media, and is one means for doing so. Closed captioningdata typically can be extracted from broadcast video or other sourcesencoded with closed captions using open source software such asCCExtractor available via SourceForge.net. For time-based media notencoded with closed captioning data, imperfect methods such as automaticspeech recognition can be used to capture and convert the audio datainto a text stream comparable to closed captioning text. This can bedone, for example, using open source software such as Sphinx 3 availablevia SourceForge.net. Once the closed captioning is ingested, it ispreferably aligned to speech in a video. Various alignment methods areknown in the art. One such method is described in Hauptmann, A. andWitbrock, M., Story Segmentation and Detection of Commercials inBroadcast News Video, ADL-98 Advances in Digital Libraries Conference,Santa Barbara, Calif. (April 1998), which uses dynamic programming toalign words in the closed captioning stream to the output of a speechrecognizer run over the audio track of the video. The closed captioninginformation is stored in the closed captioning store 267.

The multimedia store 270 stores various forms of time-based media.Time-based media includes any data that changes meaningfully withrespect to time. Examples include, and are not limited to, videos,(e.g., television programs or portions thereof, movies or portionsthereof) audio recordings, MIDI sequences, animations, and combinationsthereof. Time-based media can be obtained from a variety of sources,such as local or network stores, as well as directly from capturedevices such as cameras, microphones, and live broadcasts. It isanticipated that other types of time-based media within the scope of theinvention will be developed in the future (e.g., 3D media, holographicpresentations, immersive media, and so forth).

The event segmentation engine 220 segments time-based media intosemantically meaningful segments corresponding to discrete portions or“events,” and is one means for doing so. Different types of media mayhave different types of events which are recognized as part of a videoevent segmentation process. For example, a television program or moviemay have scenes and shots; a sporting event may have highly granularevents (e.g., plays, passes, catches, hits, shots, baskets, goals, andthe like) as well has less granular events (e.g., sides, downs, innings,and the like). A new program may have events such as stories,interviews, shots, commentary and the like. The video event segmentationprocess includes three main components according to one embodiment: shotboundary detection, event detection, and boundary determination. Thesecomponents for event segmentation may vary by domain. The output ofvideo event segmentation is a set of segmented video events that isstored in the video event store 280.

The feature extraction engine 225 converts segmented time-based mediaevents retrieved from the video event store 280 into feature vectorrepresentations for aligning the events with metadata, and is one meansfor doing so. The features may include image and audio properties andmay vary by domain. Feature types may include, but are not limited to,scale-variant feature transform (SIFT), speeded up robust features(SURF), local energy based shape histogram (LESH), color histogram, andgradient location orientation histogram (GLOH).

The metadata alignment engine 230 aligns video event segments withsemantically meaningful information regarding the event or topic thatthe event is about, and is one means for doing so. The metadataalignment engine 230 uses metadata instances from the event metadatastore 273. A metadata instance is the metadata for a single event, i.e.,a single piece of metadata. The annotation engine 235 annotates thesegments with the metadata, and is one means for doing so. Metadatainstances may include automatic annotations of low level contentfeatures, e.g., image features or content features, hand annotationswith text descriptions, or both. The metadata may be represented as textdescriptions of time-based media events and/or feature vectorrepresentations extracted from examples of events. The annotations arestored in the annotated event store 290.

The comparative feature extraction engine 240 converts an annotatedevent and a corresponding social media content item into a featurevector representation, and is one means for doing so. The three majortypes of features extracted by the comparative feature extraction engine240 are content features, geo-temporal features, and authority features.The media/event alignment engine 245 aligns the social media contentitem 610 and annotated event 530 using the extracted features 620, andis one means for doing so. The media/event alignment engine 245 outputsan annotated event/social media mapping and associated confidence scoreto the mapping store 275.

The following is a non-comprehensive list of media types that can beassociated with time-based media: audio of commentators on, orparticipants of, the event or topic (e.g., announcers on TV or radio)and text transcriptions thereof (generated manually or automatically),event-related metadata (e.g., recipes, instructions, scripts, etc.),statistical data (e.g., sports statistics or financial data streams),news articles, social media content items, and media usage statistics(e.g., user behavior such as viewing, rewind, pausing, etc.). The socialmedia content items include long form and short form social mediacontent items such as posts, videos, photos, links, status updates, blogentries, tweets, and the like from various social media and mainstreamnews sources that are stored in the social media content store 260. Ingeneral, social networks allow their users to publish text-based contentitems to other members of their network, which content items may be openand viewable by the public through open application program interfaces.

Typically social media content items are of two varieties: statictext-based media and dynamic text-based media. Static text-based mediadescribes a large class of information on the Internet (e.g., blogs,news articles, webpages, etc.). This information changes only minimallyonce posted (i.e., is relatively static) and is entirely made up ofwords (i.e., is text-based). Dynamic text-based media refer to any of aset of “data feeds” composed of short, frequently updated user posts tosocial network websites that often describe the states and opinions oftheir authors.

For some domains, usage statistics may be ingested, either alone orgenerated from the time-based media in the multimedia store 270, andstored in the usage stats store 265. Usage statistics may includeinformation regarding how the multimedia data was consumed, e.g., numberof views, length of views, number of pauses, time codes at which a pauseoccurs, etc. The statistics can be aggregated with respect to differentpopulations, such as by user type, location, usage type, media type, andso forth. The statistics can represent means, modes, medians, variances,rates, velocities, population measures, and the like.

The social interest estimator 250 aggregates information from theannotated event store 290 and the mapping store 275 to estimate socialinterest in a given media event using a social interest score, and isone means for doing so. The social interest score is estimated by thesocial interest estimator 250 by cycling through all (or selected)annotated events, and for each event, taking a weighted sum of theconfidence scores for each social media content item that exceeds agiven threshold. The resulting social interest score is stored in thesocial interest store 285.

The user interface engine 255 converts the social interest into a formatfor display on a user interface, e.g., for depicting social interestheat maps as shown in FIGS. 8A-13D, and is one means for doing so. Theuser interface engine 255 allows the client devices 150 to interact withthe user interfaces providing the social interest score.

The user interface engine 255 provides a user interface display withthree main areas: (1) a social interest heat map area for displaying asocial interest heat map showing the levels of social interest for aplurality of events corresponding to a series of chronological timesegments, (2) a media display area, visually distinguished from andconcurrently displayed with the social interest heat map area, fordisplaying an event selected from the social interest heat map, and (3)a social media display area, visually distinguished from andconcurrently displayed with the social interest heat map and mediadisplay areas, for displaying social media content items for theselected event.

Different event types may be displayed such that the different eventtypes each are visually distinct within the social interest heat maparea, e.g., for a football game on broadcast television, showing eventscorresponding to plays of the game in one manner (e.g., a first color)and events corresponding to commercials in between plays of the game ina different manner (e.g., a second color).

In addition, the user interface engine 255 may provide additionalfunctionality for the user interface. For example, a user interfacefield for filtering the time segments according to a keyword or searchterm, wherein the social interest heat map area then display only asubset of time segments matching the search term. See FIG. 11A,reference numeral 1105. In another example, the user interface may allowfor separate display of positive and negative sentiment among aggregatedcontent items for each event segment. A first portion of the segment maycorresponding to a positive sentiment, a second portion of the segmentmay correspond to a negative sentiment, and both segments may bedisplayed, such that they are visually distinguished from each other.See FIG. 10A, reference numerals 1010, 1012. In some embodiments, anadditional portion of the segment may correspond to neutral or uncertainsentiment. The domain ontology engine 205 may provide the filteringaspects for the user interface, and the social interest estimator 250may provide the sentiment analysis.

Mapping Social Media Content Items to Time-Based Media

FIG. 3 is a conceptual diagram illustrating the video/metadataalignment/annotation 320 and social media/event alignment 330 processesat a high level according to one embodiment. Beginning with metadatainstances 307 and events in time-based media 301 as input, annotatedevents 309 are formed. As shown, time-based media (TBM) 301 includesmultiple segments (seg. 1-M) 303, which contain events in the time-basedmedia, as described herein. The video/metadata alignment/annotation 320process aligns one or more metadata instances (1-N) 307 with the eventsto form annotated events 309, as further described in conjunction withFIG. 5. The social media/event alignment 330 process aligns, or “maps,”the annotated events 309 from the video/metadata alignment/annotation320 to one or more social media content items (A-O) 311, as furtherdescribed in conjunction with FIG. 6. Note that in both processes 320,330, the various alignments are one-to-one, many-to-one, and/ormany-to-many. Thus, a given social media content item 311 can be mappedto multiple different annotated events 309, and an annotated event 309can be mapped to multiple different social media content items 311. Onceso mapped, the relationships between content items and events can bequantified to estimate social interest, as further explained below.

FIG. 3A is a flow diagram illustrating one embodiment of a method foraligning social media content items (and references to events therein)with time-based media events, and a related method of determining socialinterest in the events based on the resulting associations.

Generally, social media content items are candidates for aligning withtime-based media events, and a confidence score is determined for eachindicative of a probability that the content item is relevant to theevent. Based on the confidence scores, the content items may be alignedwith the event, and the alignments are collected in a data store. Theconfidence scores are aggregated to produce an aggregate score, and alevel of social interest in the event is established based upon theaggregate score.

As a preliminary step in the method, multiple streams of data areingested 300 at the social interest information provider 130 forprocessing. Data may be received at the social interest informationprovider 130 directly from content providers, or via social mediasources 110 or time-based media sources 120, e.g., from broadcasttelevision feeds, directly from content producers, and/or from otherthird parties. In one embodiment, web server 200 is one means foringesting 300 the data. The types of data may include, but are notlimited to, time-based media, closed captioning data, statistics, socialmedia posts, mainstream news media, and usage statistics, such asdescribed above.

The ingested data is stored in data stores specific to one or more datatypes that serve as the input data sources for the primary processes ofthe method of FIG. 3A (each shown in bold). For example, time-basedmedia data is stored in the multimedia store 270. The time-based mediain the multimedia store 270 may undergo additional processing beforebeing used within the methods shown in FIGS. 3-7. For example, closedcaptioning data can be extracted from, or created for 305, thetime-based media, e.g., by closed captioning extractor 215. In addition,for some domains, usage statistics may be ingested, either alone orgenerated from the time-based media in the multimedia store 270, andstored in the usage stats store 265. In addition, event metadataassociated with multimedia is stored in the event metadata store 273,social media content items as described herein are stored in the socialmedia content store 260, information about authors of social mediacontent items are stored in the social media author store 263, anddomain ontologies indicating, for example, vocabularies specific todifferent media types, are stored in the domain ontologies 257.

As a result of the ingestion referenced above, the multimedia store 270includes various forms of time-based media. The time-based media may beof various types, as described in conjunction with FIG. 2.

As shown in FIG. 3A, there are three major processes involved in themethod according to the depicted embodiment: video event segmentation310, video metadata alignment 320, and social media/eventalignment/mapping 330. In addition, an optional process, social interestestimation 340, may be included in the method. Each of these processes310-340 are described below.

Video Event Segmentation

The first process is video event segmentation 310, in which thetime-based media is segmented into semantically meaningful segmentscorresponding to discrete events depicted in video. The input to thevideo event segmentation 310 process is a raw video (and/or audio)stream that is retrieved from the multimedia store 270 according to oneembodiment, and may be performed, e.g., by the event segmentation engine220, which is one means for performing this function.

The video event segmentation 310 process is domain dependent to someextent, e.g., in video of sporting events, event segments may be equatedwith individual plays, while in broadcast television, event segments maybe equated with individual scenes and advertisements. Thus the eventtypes and segment size may vary based on the domain type, and for somemedia, e.g., short format media such as very short video clips, theentire clip is treated as one segment. They system may be pre-configuredwith information about to which domain the video belongs. Thisconfiguration may be implemented by hand on a case by case basis, orbased on a preloaded schedule based on the source of video and time ofday (using, for example, a programming guide of broadcast televisionshows).

Segmentation may be achieved via human annotation, known automatedmethods, or a hybrid human/automatic approach in which automatic segmentboundaries are corrected by human annotators according to variousembodiments. One automated method is described in Fleischman, M. andRoy, D., Unsupervised Content-Based Indexing of Sports Video Retrieval,9th ACM Workshop on Multimedia Information Retrieval (MIR), Augsburg,Germany (September 2007).

The video event segmentation 310 process includes three main componentsaccording to one embodiment: shot boundary detection, event detection,and boundary determination. These components may vary by domain. Forexample, for sporting events an additional component may correspond toscene classification (e.g., field or stadium identification).

The output of video event segmentation 310 is a set of segmented videoevents that are stored in the video event store 280. Video eventsegmentation 310 is described in further detail in conjunction with FIG.4.

Metadata Alignment/Annotation

The next process is metadata alignment/annotation 320, in which thesegments from video event segmentation 310 are annotated withsemantically meaningful information regarding the event that the segmentis relevant to, or depicts. Input to metadata alignment/annotation 320is a video event retrieved from the video event store 280 and metadatafrom the event metadata store 273. Such metadata can include, but is notlimited to: the type of event occurring, the agents involved in theevent, the location of the event, the time of the event, theresults/causes of the event, etc.

As with event segmentation 310, the metadata alignment/annotation 320process is domain dependent. For example, in American football, metadatafor an event may include information such as “Passer: Tom Brady, Result:Touchdown, Receiver: Randy Moss,” while metadata for an event in aTelevision series may include information such as: “Agent: Jack Bauer,Location: White House, Time: 3:15 pm,” and for an advertisement themetadata may include information such as “Brand: Walmart, Scene: fatherdresses up as clown, Mood: comic.” As illustrated in these examples, themetadata can be structured as tuples of <name, value> pairs.

The metadata includes text and, for certain domains, lower level imageand audio properties. Metadata may be generated using human annotation(e.g., via human annotators watching events or samples thereof) and, incertain domains, may be supplemented with automatic annotations for usein the alignment process (e.g., describing lower level image and audioproperties of the event such as number and length of each shot, averagecolor histograms of each shot, power levels of the associated audio,etc.) The annotation is stored in the annotated event store 290.

Metadata alignment/annotation 320 includes two steps according to oneembodiment: event feature extraction and video metadata alignment.Metadata alignment/annotation 320 is described in further detail inconjunction with FIG. 5.

According to another embodiment, data ingestion 300, video eventsegmentation 310, and video metadata alignment 320 could be performed bya separate entity, such as a content provider or owner, e.g., which doesnot want to release the content to others. In this embodiment, thesocial interest information provider 130 would provide software,including the software modules and engines described herein, to theseparate entity to allow them to perform these processes on the rawtime-based media. The separate entity in return could provide the socialinterest information provider 130 with the extracted features and otherinputs to the social media/event alignment 330 and social interestestimation 340 processes, which then would be used by the socialinterest information provider 130 in those processes. These dataexchanges could take place via an application programming interface(API) provided by the social interest information provider 130 andexposed to the separate entity, e.g., via web server 200. The socialinterest information provider 130 would then compute the social interestinformation and provide that back to the entity, as either data, ordisplayed information, for example using the interfaces shown in FIGS.8A-13D.

Social Media/Event Alignment

The next step is to integrate the annotated time-based media eventsegments with social media content items that refer to the events. Inputto social media/event alignment 330 according to one embodiment is anannotated event retrieved from the annotated event store 290, a socialmedia content item retrieved from the social media content store 260, adomain ontology retrieved from the domain ontologies 257, and optionallyauthor information about the social media content item author retrievedfrom the social media author store 263.

Unfortunately, social media content items often are ambiguous as towhether they refer to an event at all, and if so, which event they referto. For example, a simple social media content item, such as the singleword post “Touchdown!” may refer to an event in a football game, or itmay be used as a metaphor for a success in areas unrelated to football.In order to address such ambiguities, the social media/event alignment330 determines a confidence score that a given social media content itemrefers to a specific event. The method takes as input a single socialmedia content item and a single annotated event, and outputs a scorerepresenting the confidence (i.e., likelihood, probability) that thesocial media content item is relevant to the event. A social mediacontent item can be relevant to an event by referring to the event. Thesocial media/event alignment 330 function operates on features of theindividual social media content items and annotated events, and can betrained using supervised learning methods or optimized by hand. Themedia/event alignment engine 245 is one means for performing thisfunction.

The output of social media/event alignment 330 is a mapping between anannotated event and a social media content item (and/or references toevents therein) and an associated confidence score. The mapping andconfidence score are stored in a mapping store 275. The socialmedia/event alignment 330 process is described in further detail inconjunction with FIG. 6.

The mappings output by social media/event alignment 330 are useful inand of themselves, as they may be used as the basis for multipleapplications, such as, ranking of search results for time-based media,automatic recommendations for time-based media, prediction of audienceinterest for media purchasing/planning, and estimation of socialinterest as described further below.

Social Interest Estimation

One of the uses of the social media/event mappings is the estimation ofsocial interest in various events. Social interest in an event may beestimated by aggregating the information gleaned from the processesdescribed with respect to FIG. 3A. The input to social interestestimation 340 is an annotated event retrieved from the annotated eventstore 290 and the annotated event social media mapping retrieved fromthe mapping store 275. In addition, inputs from the social media contentstore 260 and social media author store 263 may be used as part of theweighting process. The social interest estimator 250 is one means forperforming this function.

The social interest estimation 340 is achieved for an annotated event bycycling through all social media content items associated with thatevent (as indicated by the presence of an annotated event/social mediamapping 630 (FIG. 6) in the mapping store 275), and taking a weightedsum of the confidence scores for each social media content item. In oneembodiment, a weighted sum of the confidence scores is taken for socialmedia content items that exceed a threshold. In other embodiments, nothreshold is used or a function with a “sliding scale” of (score,weight) where the weight is applied to the score, and then added to thesum. The effect of this weighting is that the events that are associatedwith more social media content items (and references to events therein)correlate with higher estimated social interest. In addition, socialinterest in an event often is dependent on the source, author, and/orsentiment of the social media content item referencing it, as describedfurther in conjunction with weighting function 710 in FIG. 7.

The output of the social interest estimation 340 is a social interestscore that is stored in the social interest store 285. The socialinterest estimation 340 is described in further detail in conjunctionwith FIG. 7. In addition, the social interest estimation 340 results maybe displayed to a user of a social interest information device 150,e.g., using user interface engine 255, as described in conjunction withFIGS. 8A-13D.

The social interest score may be used as the basis for multipleapplications, such as data analytics, media planning, ranking of searchresults for time-based media, automatic recommendations for time-basedmedia, direct end-user data navigation via a user interface, andprediction of audience interest for media purchasing/planning to name afew.

Event Segmentation

FIG. 4 is a flow diagram illustrating one embodiment of a video eventsegmentation process 310. As described in FIG. 3A, video eventsegmentation 310 segments time-based media into semantically meaningfulsegments corresponding to discrete video portions or “events,” e.g., viaevent segmentation engine 220, which is one means for performing thisfunction.

Input to the video event segmentation process 310 is a video stream 405from the multimedia store 270. Video event segmentation 310 includes 3phases: shot boundary detection 410, event detection 420, and eventboundary determination 430, each of which is described in greater detailbelow. The output of video event segmentation 310 is a segmented videoevent 435, which is stored in the video event store 280.

Shot Boundary Detection

The first step in segmenting is shot boundary detection 410 for discretesegments (or “shots”) within a video. Shot boundaries are points ofnon-continuity in the video, e.g., associated with a change in a cameraangle or scene. Shot boundaries may be determined by comparing colorhistograms of adjacent video frames and applying a threshold to thatdifference. Shot boundaries may be determined to exist wherever thedifference in the color histograms of adjacent frames exceeds thisthreshold. Many techniques are known in the art for shot boundarydetection. One exemplary algorithm is described in Tardini et al., ShotDetection and Motion Analysis for Automatic MPEG-7 Annotation of SportsVideos, 13th International Conference on Image Analysis and Processing(November 2005). Other techniques for shot boundary detection 410 may beused as well, such as using motion features. Another known technique isdescribed in A. Jacobs, et al., Automatic shot boundary detectioncombining color, edge, and motion features of adjacent frames, Centerfor Computing Technologies, Bremen, Germany (2004).

Event Detection

Event detection 420 identifies the presence of an event in a stream of(one or more) segments using various features corresponding, forexample, to the image, audio, and/or camera motion for a given segment.A classifier using such features may be optimized by hand or trainedusing machine learning techniques such as those implemented in the WEKAmachine learning package described in Witten, I. and Frank, E., DataMining: Practical machine learning tools and techniques (2nd Edition),Morgan Kaufmann, San Francisco, Calif. (June 2005). The event detectionprocess 420 details may vary by domain.

Image features are features generated from individual frames within avideo. They include low level and higher level features based on thosepixel values. Image features include, but are not limited to, colordistributions, texture measurements, entropy, motion, detection oflines, detection of faces, presence of all black frames, graphicsdetection, aspect ratio, and shot boundaries.

Speech and audio features describe information extracted from the audioand closed captioning streams. Audio features are based on the presenceof music, cheering, excited speech, silence, detection of volume change,presence/absence of closed captioning, etc. According to one embodiment,these features are detected using boosted decision trees. Classificationoperates on a sequence of overlapping frames (e.g., 30 ms overlap)extracted from the audio stream. For each frame, a feature vector iscomputed using Mel-frequency cepstral coefficients (MFCCs), as well asenergy, the number of zero crossings, spectral entropy, and relativepower between different frequency bands. The classifier is applied toeach frame, producing a sequence of class labels. These labels are thensmoothed using a dynamic programming cost minimization algorithm,similar to those used in hidden Markov models.

In addition to audio features, features may be extracted from the wordsor phrases spoken by narrators and/or announcers. From a domain specificontology 257, a predetermined list of words and phrases is selected andthe speech stream is monitored for the utterance of such terms. Afeature vector representation is created in which the value of eachelement represents the number of times a specific word from the list wasuttered. The presence of such terms in the feature vector correlateswith the occurrence of an event associated with the predetermined listof words. For example, the uttering of the phrase “touchdown” iscorrelated with the occurrence of a touchdown in sports video.

Unlike image and audio features, camera motion features represent moreprecise information about the actions occurring in a video. The cameraacts as a stand in for a viewer's focus. As actions occur in a video,the camera moves to follow it; this camera motion thus mirrors theactions themselves, providing informative features for eventidentification. Like shot boundary detection, there are various methodsfor detecting the motion of the camera in a video (i.e., the amount itpans left to right, tilts up and down, and zooms in and out). Oneexemplary system is described in Bouthemy, P., et al., A unifiedapproach to shot change detection and camera motion characterization,IEEE Trans. on Circuits and Systems for Video Technology, 9(7) (October1999); this system computes the camera motion using the parameters of atwo-dimensional affine model to fit every pair of sequential frames in avideo. According to one embodiment, a 15-state first-order hidden Markovmodel is used, implemented with the Graphical Modeling Toolkit, and thenthe output of the Bouthemy is output into a stream of clusteredcharacteristic camera motions (e.g., state 12 clusters together motionsof zooming in fast while panning slightly left). Some domains may usedifferent, or additional, methods of identifying events. For example, inAmerican football, an additional factor may be scene classification. Inscene classification, once a shot boundary is detected a sceneclassifier is used to determine whether that shot is primarily focusedon a particular scene, e.g., a playing field. Individual frames (calledkey frames) are selected from within the shot boundaries and representedas a vector of low level features that describe the key frame's colordistribution, entropy, motion, etc. A shot is determined to be of aparticular scene if a majority of the sampled frames is classified asthat scene.

Event Boundary Determination

Once a segment of video is determined to contain the occurrence of anevent, the beginning and ending boundaries of that event must bedetermined 430. In some cases, the shot boundaries determined in 410 areestimates of the beginning and end of an event. The estimates can beimproved as well by exploiting additional features of the video andaudio streams to further refine the boundaries of video segments. Eventboundary determination 430 may be performed using a classifier that maybe optimized by hand or using supervised learning techniques. Theclassifier may make decisions based on a set of rules applied to afeature vector representation of the data. The features used torepresent video overlap with those used in the previous processes.Events have beginning and end points (or offsets), and those boundariesmay be determined based on the presence/absence of black frames, shotboundaries, aspect ratio changes, etc., and have a confidence measureassociated with the segmentation. The result of event boundarydetermination 430 (concluding video event segmentation 410) is a (setof) segmented video event 435 that is stored in the video event store280.

Metadata Alignment/Annotation

FIG. 5 is a flow diagram illustrating one embodiment of a metadataalignment/annotation 320 process. As described in FIG. 3A, the metadataalignment/annotation 320 process produces annotations of the segmentsfrom video event segmentation 310, which annotations includesemantically meaningful information regarding the event or topic thatthe segment is about. Metadata alignment/annotation 320 includes twosteps: event feature extraction 315 and video metadata alignment 520.

Video Feature Extraction

For any given video event that is to be aligned with metadata, the firststep is to convert the video event into a feature vector representationvia feature extraction 315. The feature extraction engine 225 is onemeans for performing this function. Input to the process is a segmentedvideo event 435 retrieved from the video event store 280. Output fromthe video feature extraction 315 is a video event feature representation510. The features may be identical to (or a subset of) the image/audioproperties discussed above for video events and stored in the eventmetadata store 273, and may vary by domain.

Video Metadata Alignment

Video metadata alignment 520 takes as input the feature vectorrepresentation 510 of an event and a metadata instance 505, definedabove as metadata corresponding to a single event. The metadataalignment engine 230 is one means for performing this function. Itcycles through each metadata instance 505 in the event metadata store273 and uses an alignment function to estimate the likelihood that aparticular event may be described by a particular metadata instance foran event. As described above, metadata instances may include automaticannotations of low level content features (e.g., image or audiofeatures), hand annotations of text descriptions, or both. For domainsin which the metadata includes low level features, the alignmentfunction may be a simple cosign similarity function that compares thefeature representation 510 of the event to the low level propertiesdescribed in the metadata instance 505. For domains in which metadatainstances do not include automatic annotations of low level features,the video metadata alignment 520 method may employ a model which encodesrelationships between low level features and descriptive text. Oneexemplary model is described in Fleischman, M. and Roy, D., GroundedLanguage Modeling for Automatic Speech Recognition of Sports Video,Proceedings of the Association of Computational Linguistics (ACL),Columbus, Ohio, pp. 121-129 (June 2008). This method uses groundedlanguage models that link visual and text features extracted from avideo to the metadata terms used to describe an event. For the purposesof this example, grounded language models can be manually estimatedbased on the visual and text features used for event segmentation, fromwhich the following equation describes the likelihood that anyparticular metadata annotation describes a particular video event:

$p( {{{metadata} {Vid} )} = {\prod\limits_{w \in \;{metadata}}^{\;}\;{\sum\limits_{v \in {Vid}}^{\;}{{p( w }v}}}} )$The grounded language model is used to calculate the probability thateach video event found is associated with each human generated metadataannotation.

When all metadata instances 505 in the event metadata store 273corresponding to the event have been examined, if the most likelyalignment 525 (i.e., alignment with the highest probability or score)passes a threshold, the video event associated with the featurerepresentation 510 is annotated with the metadata instance 505 and theresulting annotated event 530 is stored in an annotated event store 290along with a score describing the confidence of the annotation. If noevent passes the threshold, the event is marked as not annotated. Inorder to set this threshold, a set of results from the process is handannotated into two categories: correct and incorrect results.Cross-validation may then be used to find the threshold that maximizesthe precision/recall of the system over the manually annotated resultset.

Social Media/Event Alignment

FIG. 6 is a flow diagram illustrating one embodiment of a socialmedia/event alignment 330 process. Social media/event alignment 330associates (maps) the annotated time-based media event segments withsocial media content items and references to the events therein.

Filtering

As an initial and optional step, social media filtering step 605 occurs;the domain ontologies 257 are one means for performing this function.Social media content items are filtered in order to create a set ofcandidate content items with a high likelihood that they are relevant toa specific event. Content items can be relevant to an event by includinga reference to the event.

In this optional step, before social media content items are integratedwith video events, a candidate set of content items is compiled based onthe likelihood that those posts are relevant to the events, for example,by including at least one reference to a specific event. The comparativefeature extraction engine 240 is one means for performing this function.At the simplest, this candidate set of content items can be the resultof filtering 605 associated with a given time frame of the event inquestion. Temporal filters often are far too general, as many contentitems will only coincidentally co-occur in time with a given event. Inaddition, for broadcast television, e.g., the increasing use of digitalvideo recorders has broadened significantly the relevant timeframe forevents.

Additional filters 605 are applied based on terms used in the contentitem's text content (e.g., actual texts or extracted text from closedcaption or audio) that also appear in the metadata for an event and/ordomain specific terms in the ontologies 257. For example, content itemof a social network posting of “Touchdown Brady! Go Patriots” has a highprobability that it refers to an event in a Patriots football game dueto the use of the player name, team name, and play name, and thiscontent item would be relevant to the event. In another example, acontent item of a post that “I love that Walmart commercial” has a highprobability that it refers to an advertisement event for Walmart due tothe use of the store name, and the term “commercial,” and thus wouldlikewise be relevant to this event. To perform this type of filtering,terms are used from the metadata of an event as well as thosedomain-specific terms stored in ontology 257.

A social media content item can be relevant to an event withoutnecessarily including a direct textual reference to the event. Variousinformation retrieval and scoring methods can be applied to the contentitems to determine relevancy, based on set-theoretic (e.g., Booleansearch), algebraic (e.g., vector space models, neural networks, latentsemantic analysis), or probabilistic models (e.g., binary independence,or language models), and the like.

Social media content items that do not pass certain of these initialfilters, e.g., temporal or content filters, are removed from furtherprocessing, reducing the number of mappings that occur in the lattersteps. The output of social media filtering 605 is an updated socialmedia content store 260, which indicates, for each content item, whetherthat content item was filtered by temporal or content filters.Additional filters may apply in additional domains.

Alignment/Mapping

Social media/annotated event alignment 330 includes a feature extractionprocess 620 and an alignment function 625. The feature extractionprocess 620 converts input of an annotated event 530 and a social mediacontent item 610 into a feature vector representation, which is theninput to the alignment function 625. The feature extraction process 620also may receive input from the social media author store 263 and thedomain ontologies 257. The three major types of features extracted inthis process 620 are content features 620 c, geo-temporal features 620b, and authority features 620 a. The comparative feature extractionengine 240 is one means for performing this function, which identifies arelationship between the event features and social media features. Therelationship may be co-occurrence, correlation, or other relationshipsas described herein.

Content features 620 c refer to co-occurring information within thecontent of the social media content items and the metadata for the videoevents, e.g., terms that exist both in the content item and in themetadata for the video event. Domain ontologies 257 may be used toexpand the set of terms used when generating content features.

Geo-temporal features 620 b refer to the difference in location and timeat which the input media was generated from a location associated withthe social media content item about the event. Such information isuseful as the relevance of social media to an event is often inverselycorrelated with the distance from the event (in time and space) that themedia was produced. In other words, social media relevant to an event isoften produced during or soon after that event, and sometimes by peopleat or near the event (e.g., a sporting event) or exposed to it (e.g.,within broadcast area for television-based event).

For video events, geo-temporal information can be determined based onthe location and/or time zone of the event or broadcast of the event,the time it started, the offset in the video that the start of the eventis determined, the channel on which it was broadcast. For social media,geo-temporal information can be part of the content of the media itself(e.g., a time stamp on a blog entry or status update) or as metadata ofthe media or its author.

The temporal features describe the difference in time between when thesocial media content item was created from the time that the eventitself took place. In general, smaller differences in time of productionare indicative of more confident alignments. Such differences can bepassed through a sigmoid function such that as the difference in timeincreases, the probability of alignment decreases, but plateaus at acertain point. The parameters of this function may be tuned based on anannotated verification data set. The spatial features describe thedistance from the author of the content item location relative to thegeographical area of the event or broadcast. Spatial differences areless indicative because often times people comment on events that takeplace far from their location. A sigmoid function may be used to modelthis relationship as well, although parameters are tuned based ondifferent held out data.

Authority features 620 a describe information related to the author ofthe social media and help to increase the confidence that a social mediacontent item refers to a video event. The probability that any ambiguouspost refers to a particular event is dependent upon the priorprobability that the author would post about a similar type of event(e.g., a basketball game for an author who has posted content aboutprior basketball games). The prior probability can be approximated basedon a number of features including: the author's self-generated userprofile (e.g., mentions of a brand, team, etc.), the author's previouscontent items (e.g., about similar or related events), and the author'sfriends (e.g., their content contributions, profiles, etc.). These priorprobability features may be used as features for the mapping function.

The alignment function 625 takes the set of extracted features 620 a-cand outputs a mapping 630 and a confidence score 640 representing theconfidence that the social media content item refers to the video event.The media/event alignment engine 245 is one means for performing thisfunction. For each feature type 620 a-c, a feature specific sub-functiongenerates a score indicating whether the social media content itemrefers to the annotated event. Each sub-function's score is based onlyon the information extracted in that particular feature set. The scoresfor each sub-function may then be combined using a weighted sum, inorder to output a mapping 630 and an associated confidence score 640, asshown below for an event x and a social media content item y:align(feat(x,y))=[α·content(feat(x,y))]+[β·geoTemp(feat(x,y))]+[γ·author(feat(x,y))]

where α, β, and γ are the respective weights applied to the threefeature types, and align(feat(x,y)) is the confidence score. Both theweights in the weighted sum, as well as the sub-functions themselves maybe trained using supervised learning methods, or optimized by hand. Theoutput of the social media/event alignment function 330 is a mappingbetween an annotated event and a social media content item. Thismapping, along with the real-value confidence score is stored in themapping store 275.

Social Interest Estimation

FIG. 7 is a flow diagram illustrating one embodiment of a socialinterest estimation process 340. Social interest in an event may beestimated by aggregating the information gleaned from the video eventsegmentation 310, video metadata alignment 320, and social media/eventalignment 330 processes. The social interest estimator 250 is one meansfor performing this function.

Input to the social interest estimation process 340 includes anannotated event 530 retrieved from the annotated event store 290 and anannotated event/social media mapping 620 retrieved from the mappingstore 275. In addition, data from the social media content store 260 andsocial media author store 263 may be used for the weighting function710.

For each of the media types, social interest is estimated based on aweighted count of references to particular events in each social mediacontent item. Social media content items relevant to an event areindicative of interest, and by discovering and aggregating such contentitems and references to events therein, a social interest score isgenerated that represents the level of social interest of the eventbased on the aggregated content items.

For a particular event, the social interest estimation process 340includes the computation of a weighted sum over all social media contentitems that include at least one reference to an event. The computationproceeds by cycling through all social media content items that refer tothat event (as determined in the social media/annotated event alignment330 process). For each item aligned to that event the social interestscore for that event is incremented by a weighted value based on themetadata of the content item. Thus, the output social interest score 720can be thought of as an aggregate score aggregated across the confidencescores 640 for each event.

These weights typically can be set from zero to one depending on theconfiguration of the system. The weights are multiplicative, and arebased on various factors described below: as social media contentweights 710 a, source-based weights 710 b, author-based weights 710 c,and/or event-based weights 710 d.

Social media content weights 710 a can be used in the social interestestimation process 340 based on, for example, the sentiment of the mediathat mention it. For example, scores can be weighted such that interestis computed based only on posts that describe positive sentiment towardan event (i.e., only posts from authors who expressed positive sentimenttoward the event are incorporated in the weighted sum). The sentimentexpressed in a social media content item may be automatically identifiedusing a number of techniques. Exemplary techniques are described in B.Pang and L. Lee, Opinion Mining and Sentiment Analysis, Foundations andTrends in Information Retrieval 2(1-2), pp. 1-135 (2008).

Source-based weights 710 b can be used in the social interest estimationprocess 340 based on how (e.g., in what form) an event is mentioned.Some sources may be given higher weight if they are determined to bemore influential as measured by, for example, the size of their audience(as estimated, for example, by QuantCast Corporation, San Francisco,Calif.) or the number of inbound links to the source site. Further,certain sources may be given higher weight in order to generate socialinterest scores for specific communities of users. For example, a socialinterest score may be computed based on only social media content itemsgenerated by sources of a particular political leaning (e.g., Republicanor Democrat) by setting the weights to zero of all content items withsources that are not predetermined to be of that particular politicalleaning (e.g., where the political leaning of a source is determined bya human expert or a trained machine classifier).

Author-based weights 710 c can be used in the social interest estimationprocess 340 to bias the social interest estimate toward specificcommunities of users. For example, the estimate of social interest maybe biased based on demographic information about the author of the post,such that, for example, only posts that were generated by men older than25 years old are given weight greater than zero. Determination of suchdemographic information may come from an examination of publiclyavailable data posted by the author themselves, by human annotation ofspecific authors based on expert opinion, or by machine classifierstrained on human labeled examples. In the sports context, estimate ofsocial interest can be weighted toward only fans of the home team byfiltering posts based on their location of origin (i.e. only posts fromauthors in the home team's city are incorporated in the weighted sum) orprevious history of posts (i.e. the author has a history of postingpositive remarks about the home team).

Event-based weights 710 d can be used in the social interest estimationprocess 340 based on evidence of social interest within the time-basedmedia stream itself. Examples of such media include, but are not limitedto, series television shows, and broadcast sports games. In suchtime-based media, multiple features exist that provide informationuseful for estimating social interest. Examples of this include, but arenot limited to, visual analysis (e.g., looking for specific events, suchas explosions), audio analysis (e.g., identification of high energysound events, such as excited speech), natural language analysis (e.g.identification of key terms and phrases, such as “home run”), and videoevent analysis (e.g., evaluation of replayed events such as those shownat the beginning of series television shows or intermittently in sportsbroadcasts such as an instant replay in a sporting event). Weights basedon such events themselves are predetermined using analysis of humanlabeled examples.

Further, the social interest scores can be weighted based on thebehaviors of viewers of the time-based media, as stored in the usagestatistics 265. Such user behavior is integrated based upon the timingof user content items relative to media and presentation times of theevents (e.g., how often a particular event was replayed). Analysis ofthese behaviors across multiple users can be indicative of socialinterest, for example, when the same section of media is paused andreviewed multiple times (by multiple people). Other recordable userbehavior from the usage statistics 265 that can be used for the socialinterest estimation process 340 includes, but is not limited to, viewingtimes, dwell times, click through rates for advertisements, searchqueries, sharing behavior, etc.

The output of the social interest estimation process 340 is a socialinterest score 720 that is stored in the social interest store 285. Thesocial interest score 720 may be used to provide information for a userinterface, e.g., as described in the displays depicted herein, via userinterface engine 255, which is one means for performing this function.

To further illustrate the methods for associating social media contentitems with time-based media events, and for determining social interestin the events based on the resulting associations, two examples followin the domains of American football and commercial advertising.

Example American Football

As described in conjunction with FIG. 3A, multiples streams of data areingested as a preliminary step in the method. For the football domain,in addition to the data discussed in FIG. 3, an additional source ofdata comes from statistical feeds that contain detailed metadata aboutevents (with text descriptions of those events) in a football game.Statistical feed are available from multiple sources such as the NFL'sGame Statistics and Information System and private companies such asStats, Inc.

Video Event Segmentation

In the video event segmentation 310 process for American football, thetime-based media, e.g., a broadcast television feed for a football game,is segmented into semantically meaningful segments corresponding todiscrete “events” that include plays in a game (and advertisements inbetween).

The first step in segmenting events in a football video is to detect theshot boundaries of a video. Shot boundaries are points in a video ofnon-continuity, often associated with the changing of a camera angle ora scene. In the domain of American football, changes in camera anglesare typically indicative of changes in plays.

In the football domain, event detection 420 may operate by firstidentifying shots that depict the football field. Once a shot boundaryis detected, a scene classifier is be used to determine whether thatshot is primarily focused on the playing field. Field shots may then befurther classified as depicting a game event (i.e. a play). In thefootball domain, during event boundary determination 430 the beginningand end points (i.e., in/out points) of an event may be refined toreflect more appropriate start and stop points of a play. Such in/outpoints may be adjusted based on clock characterization, and/or utterancesegmentation. In a professional football game, the beginning and end ofa play is sometimes (but not always) associated with the starting orstopping of the play clock. This play clock is often shown as a graphicoverlay in a broadcast football game. The starting/stopping of this playclock can be determined by monitoring the amount of change (in pixels)of a frame sub-region (i.e., the region containing the play clockgraphic) in the video over time. When the aggregate change in suchsub-regions falls below a threshold for greater than one second, thestate of the play-clock is assumed to be “inactive.” If the aggregatechange goes above a threshold, the state of the play-clock is assumed tobe “active.” Changes in the state of the play-clock are strongindicators that an event has either begun or ended in the video.

Aesthetic judgment is often required when determining boundaries for theprecise start and end points of a play. Approximating such judgments canbe accomplished using the utterance boundaries in the speech of the gameannouncers. These utterances boundaries can be detected by identifyingpauses in the stream of speech in the video. Pauses can be identifiedusing audio processing software, such as is found in Sphinx 3.

Thus, the output of video event segmentation 310 for an Americanfootball game on broadcast television is a set of segmented video eventscorresponding to plays in a game.

Video Metadata Alignment/Annotation

The process of metadata alignment/annotation 320 in American footballoperates on the video stream segmented into events based on plays in thegame. These events are annotated with metadata concerning the type ofevent shown (e.g. “touchdown”), key players in those events (e.g. “TomBrady”), the roles of those players (e.g. “Passer”), and, details of theevent (e.g. “number of yards gained”). This metadata can be addedmanually by human experts, fully automatically by a machine algorithm,or semi-automatically using a human-machine hybrid approach. Metadata isstored in the event metadata store 273.

For each event (i.e., play) that is to be aligned with metadata, theplay is converted into a feature vector representation via featureextraction 315. Video metadata alignment 520 then takes as input thefeature vector representation 510 of a single play and a metadatainstance 505. It cycles through each metadata instance 505 in the eventmetadata store 273 and estimates the likelihood that the particular playmay be described by a particular metadata instance using, for example, aprobabilistic model. One exemplary model is the grounded language modeldescribed above.

Social Media/Annotated Event Alignment

In social media/annotated event alignment 330, feature extraction 620generates geo-temporal features, content features, and authorityfeatures. Content feature representations express the amount ofcorrelated content between event metadata and terms within social mediacontent items. For example, the content item “Touchdown Brady! GoPatriots,” and the annotation “passer: Brady, event: touchdown,receiver: Moss” have overlapping content terms (i.e., “touchdown” and“Brady”).

In addition to exact matches, the domain ontology 257 of football termsis used to expand the term set to include synonyms and hypernyms (e.g.,“TD” or “score” for “touchdown”), as well as nicknames for players (e.g.“Tom Terrific” for “Brady”).

Authority feature representations express the prior probability that anyauthor of social media content may be referring to a football event. Onefactor in the estimation of this probability may be based on thefriends, followers, or other connections to a user in their socialnetwork. Such connections are indicative of an author's likelihood topost about a football event, which can provide additional features forthe social media/event alignment 330 function. The more friends someonekeeps who post about football events, the more likely they will postabout football events. To capture this information, meta-scores aregenerated for a user based on the frequency that their contacts haveposted about football events. The meta-scores are the average, mode, andmedian of all of the frequency of their friends' football posts.

The output of social media/event alignment 330 is a mapping between theannotated play and each social media content item, with an associatedconfidence score.

If information about the social interest in the play is desired, it maybe estimated by aggregating the information gleaned from the aboveprocesses. The social interest estimation 340 may be calculated forevery play in the game. The likely result is higher social interestscores for plays such as touchdowns, and lower social interest scoresfor lesser plays.

Example Advertising

As described in conjunction with FIG. 3A, multiples streams of data areingested as a preliminary step in the method.

Video Event Segmentation

For the advertising domain, during the video event segmentation 310process, the time-based media is segmented into semantically meaningfulsegments corresponding to discrete “events” which are identified withadvertisements (i.e. commercials).

Event detection 420 in the advertising domain may operate by identifyingone or more shots that may be part of an advertising block (i.e. asequence of commercials within or between shows). Advertising blocks aredetected using image features such as the presence of all black frames,graphics detection (e.g. presence of a channel logo in the frame),aspect ratio, shot boundaries. Speech/audio features may be usedincluding detection of volume change, and the presence/absence of closedcaptioning.

Event boundary detection 430 operates on an advertisement block andidentifies the beginning and ending boundaries of individual ads withinthe block. Event boundary determination may be performed using aclassifier based on features such as the presence/absence of blackframes, shot boundaries, aspect ratio changes. Classifiers may beoptimized by hand or using machine learning techniques.

Video Metadata Alignment/Annotation

As with event segmentation 310, the video metadata alignment/annotation320 process is domain dependent. In the advertisement domain, metadatafor an advertisement may include information such as “Brand: Walmart,Scene: father dresses up as clown, Mood: comic.” This metadata isgenerated by human annotators who watch sample ad events and logmetadata for ads, including, the key products/brands involved in the ad,the mood of the ad, the story/creative aspects of the ad, theactors/celebrities in the ad, etc.

Metadata for advertisements may also include low level image and audioproperties of the ad (e.g. number and length of shots, average colorhistograms of each shot, power levels of the audio, etc.).

For each event (i.e., advertisement) that is to be aligned withmetadata, the advertisement is converted into a feature vectorrepresentation via feature extraction 315. Video metadata alignment 520then takes as input the feature vector representation 510 of a singleadvertisement and a metadata instance 505. It cycles through eachmetadata instance 505 in the event metadata store 273 and estimates thelikelihood that the particular advertisement may be described by aparticular metadata instance using, for example, a simple cosignsimilarity function that compares the low level feature representationof the ad event to the low level properties in the metadata.

Social Media/Annotated Event Alignment

In social media/annotated event alignment 330, feature extraction E.g.,c generates geo-temporal features, content features, and authorityfeatures. Content feature representations express the amount ofco-occurring content between event metadata and terms within socialmedia content items. For example, the content item “I loved thathilarious Walmart clown commercial” and the annotation “Brand: Walmart,Scene: father dresses up as clown, Mood: comic” have co-occurringcontent terms (i.e., “Walmart” and “clown”).

In addition to exact matches, the domain ontologies 257 that encodeinformation relevant the advertising domain may be used to expand theterm set to include synonyms and hypernyms (e.g., “hilarious” for“comic”), names of companies, products, stores, etc., as well as,advertisement associated words (e.g., “commercial”).

The output of social media/event alignment 330 is a mapping between theannotated advertisement and each social media content item, with anassociated confidence score.

If information about social interest in the advertisement is desired, itmay be estimated by aggregating the information gleaned from the aboveprocesses. The social interest estimation 340 may be calculated forevery advertisement in an advertising block or television show. Thelikely result is higher social interest scores for particularlyinteresting or funny advertisements, and lower social interest scoresfor less exciting or repetitive advertisements.

Although American football and advertising domains are described above,the methods described herein can be adapted to any domain usingtime-based media. The method of adaptation is general across differentdomains and focuses on two changes. First, techniques and features usedfor event segmentation and annotation are adapted to reflect domainspecific characteristics. For example, detecting events in footballexploits the visibility of grass as it is represented in the colordistributions in a video frame, while detecting events in news video mayexploit clues in the closed captioning stream and graphic overlays inthe frames. The second change involves the ontology used to link eventsto social media content items which refer to them. While for football,the requisite ontology contains concepts related to football players,teams, and events, domains such as news video require ontologies withconcepts related to germane concepts such as current events andculturally popular figures.

Display of Social Interest Estimation

As mentioned above, the social interest estimations can be used invarious ways. One such application is to display social interest invarious user interfaces and graphic representations. FIGS. 8A and 8Bshow two embodiments of social interest heat maps 810, 820 showinglevels of social interest for a plurality of events corresponding to aseries of chronological time segments in a time-based medium.

FIG. 8A shows a social interest heat map 810 corresponding to a footballgame, in which individual events (plays and advertisements) 815 areshown as vertical bars chronologically across a timeline 830; the timelocation of a bar corresponds to the beginning point of the event. Thelevel (height) of estimated social interest in each event 815 is shownvertically by number of social content items 850 corresponding to eachevent 815, with a taller bar representing greater social interest. Twoevent types, advertisements 870 and plays 880, are shown.

FIG. 8B shows a similar social interest heat map corresponding to afootball game, in which individual events (plays and advertisements) 860are shown chronologically across a timeline 840. The level of estimatedsocial interest in each event 860 is shown by intensity of color of thecorresponding bar 860, with a darker bar representing greater socialinterest. Other color/intensity/texture/pattern scales can be used torepresent the level of interest. Two event types, advertisements 890 andplays 860, are shown.

FIGS. 13A-D show yet another embodiment of a user interface 1300displaying social interest heat maps 1310 showing levels of socialinterest for a plurality of events corresponding to a series ofchronological time segments in a time-based medium.

FIG. 13A shows a user interface 1300 a with each social interest heatmap 1310 (horizontal bars) corresponding to a different channel. Thewidth of the maps 1310 corresponds to a time period as show in thenavigation bar 1320, between the two ends 1325. Channels have multipledistinct shows, shown as cells 1315, thereby forming a grid. The levelof social interest is indicated by intensity of color in a given cell1315, with the darkest cells indicative of the highest social interestin the show. The navigation bar 1320 allows the user to select thetimeframe for viewing, and the ends 1325 allow the size of thenavigation bar to be expanded to adjust the visible portion of thesocial interest heat maps in the user interface 1300, with the left end1325 a controlling the beginning time and the right end 1325 bcontrolling the ending time for the social interest heat maps 1310.

FIG. 13B shows a user interface 1300 b similar to that shown in FIG.13A, except that the social interest heat maps 1310 include indicationof advertisements 1330 that appear during the shows 1315. The darknessof the lines corresponding to individual advertisements with thedarkness as an indicator of social interest in the advertisements, withdarker indicating greater interest.

FIG. 13C shows a user interface 1300 c similar to that shown in FIG.13A, except that the social interest heat maps 1310 are zoomed out tothe level of days to show a different time scale on the navigation bar1337. Here, each division 1340 in the navigation bar corresponds to asingle day. The cells 1345 correspond to times of day, e.g., Primetime.The darkness of color of each cell is representative of the socialinterest in shows and/or advertisements during that time frame.

FIG. 13D shows a user interface 1300 d similar to that shown in FIG.13A, except that the social interest heat maps 1310 are zoomed out tothe level of months to show a different time scale. The division 1365 inthe navigation bar 1337 corresponds to a quarter of a year. The cells1355 in the grid correspond to months of the year. The darkness of colorof each cell is representative of the social interest in shows and/oradvertisements during that time frame.

FIGS. 9A-9C show three embodiments of user interfaces 900 of a socialinterest estimation system. Each figure shows a social interest heat maparea 910, media display area 920, and a social media display area 930(not shown in 9C).

FIG. 9A shows in the social interest heat map area 910 a three socialinterest heat maps 940 a-c similar to the one described in conjunctionwith FIG. 8A, each map 940 a-c corresponding to a different channel ofmedia content. The media display area 920 a shows a media player fordisplaying the time-based media associated with the selected event 915,in this example a Dodge Charger advertisement. The social media displayarea 930 a shows statistical information about the social mediacorresponding to the selected event, as well as the individual socialmedia content items.

FIG. 9B shows in the social interest heat map area 910 b several socialinterest heat maps 960 similar to the one described in conjunction withFIG. 8B, each map 960 corresponding to a different channel, as well asan overall social interest heat map 970 corresponding to a selectedevent across all channels. The media display area 920 b shows a mediaplayer for displaying the time-based media associated with a userselected event 935, in this example an advertisement scene. The user canselect any event 935 in the display and invoke the player to show thevideo content of the event. The social media display areas 930 b 1 and930 b 2 show the individual social media content items (930 b 1) andstatistical information about the social media corresponding to theselected event (930 b 2).

FIG. 9C shows in the social interest heat map area 910 c four socialinterest heat maps 950 a-d similar to the one described in conjunctionwith FIG. 8B, each map 950 a-d corresponding to a different channel. Themedia display area 920 c shows a media player for displaying thetime-based media associated with the selected event 925, in this examplea pass in a football game. Again, the user can control the player toshow an event by selecting the event 925 in a map 950.

FIGS. 10A and 10B show two embodiments of user interfaces 1000 of asocial interest estimation system showing a sentiment view. The userinterfaces 1000 are similar to those shown in FIGS. 9A-9B, except thatthe social interest heat maps 940, 970 provide information indicatingthe sentiment of the social media content items, i.e., whether they arenegative or positive, e.g., based on the sentiment detection processdescribed herein.

FIG. 10A shows for the event 915 in the social interest heat maps 940, a(top) positive portion 1010 corresponding to the number of social mediacontent items with positive sentiment, and a (bottom) negative portion1012 corresponding to the number of social media content items withnegative sentiment. The positive 1010 and negative 1012 portions arevisually distinguished from each other, such that their relativepercentages within the whole of the event bar is visible. A radio button1015 is shown for toggling on and off the sentiment view.

FIG. 10B shows for an event 1015 in the overall social interest heat map970, a (top) positive portion 1020 corresponding to the number of socialmedia content items with positive sentiment, and a (bottom) negativeportion 1022 corresponding to the number of social media content itemswith negative sentiment. The positive 1020 and negative 1020 portionsare visually distinguished from each other, such that their relativepercentages within the whole of the event bar is visible.

FIGS. 11A-11C show three embodiments of user interfaces 1100 of a socialinterest estimation system showing a filtered view. The user interfaces1100 are similar to those shown in FIGS. 9A-9C, except that the socialinterest heat maps 940, 970 provide information for only a filteredsubset of the social media content items.

FIG. 11A shows a text-based filter “doritos” applied to the data suchthat social media content item bars corresponding to Doritos brandadvertisements (1110) show up darker, or otherwise visuallydistinguished, from the non-Doritos brand social media content item bars(1115).

FIG. 11B shows a text-based filter applied to the data (not shown) suchthat only social media content item bars corresponding to the appliedfilter are visible in the overall social interest heat map 970.

FIG. 11C shows a filter applied to the data corresponding to players inthe user's fantasy football league, such that only social media contentitem bars corresponding to plays by the fantasy football players areshown in the social interest heat maps 950. An additional players area1120 shows the players in the user's fantasy football league.

FIG. 12A shows one embodiment of user interface 1200 of a socialinterest estimation system showing a focused unexpanded view. The userinterface 1200 is similar to that of FIG. 10A, except that the socialinterest heat map 940 a has a subsection 1210 of the social interestheat map selected. FIG. 12B shows a user interface 1250 similar to thatof FIG. 12A, except that it shows a zoom view 1260 of the socialinterest heat map 940 a with the subsection 1210 from FIG. 12A expanded.

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be persistently stored in a non-transitory, tangible computerreadable storage medium, or any type of media suitable for storingelectronic instructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

What is claimed is:
 1. A non-transitory computer readable medium storinga computer program executable by a processor, the computer programproducing a user interface of a social interest estimation system, theuser interface comprising: a media display area for displaying a mediaevent from a time-based media source, the media event having a time ofoccurrence; a social media display area, visually distinguished from andconcurrently displayed with the media display area, for displaying aplurality of relevant social media content items, the social mediacontent items: broadcasted by a social media source, the social mediasource independent from the time-based media source; and determined tobe relevant to the displayed media event based on a relationship betweenthe time of occurrence and a time of authorship of each of the pluralityof social media content items; and a social interest heat map area,visually distinguished from and concurrently displayed with the mediadisplay area and the social media display area, for displaying a socialinterest heat map showing levels of social interest for a plurality ofmedia events from the time-based media source based on a number ofsocial media content items determined to be relevant to each of themedia events.
 2. The computer readable medium of claim 1, wherein thesocial interest heat map area displays different media event types suchthat the different media event types each are visually distinct.
 3. Thecomputer readable medium of claim 1, wherein the computer program isfurther configured to determine a confidence score indicative of aprobability that the social media content items are relevant to themedia event.
 4. The computer readable medium of claim 3, whereindetermining the confidence score indicative of the probability that thesocial media content items are relevant to the media event furthercomprises: extracting media event features from annotations associatedwith the media event; extracting social media features from theplurality of social media content items; and mapping the media event tothe social media content items based on a relationship between the mediaevent features and social media features.
 5. The computer readablemedium of claim 4, further comprising annotating the media event withthe annotations using metadata instances relevant to the media event. 6.A system for producing a user interface of a social interest estimationsystem, the comprising: means for displaying a media display areadisplaying a media event from a time-based media source, the media eventhaving a time of occurrence; means for displaying a social media displayarea, visually distinguished from and concurrently displayed with themedia display area, and displaying a plurality of relevant social mediacontent items, the social media content items: broadcasted by a socialmedia source, the social media source independent from the time-basedmedia source; and determined to be relevant to the displayed media eventbased on a relationship between the time of occurrence and a time ofauthorship of each of the plurality of social media content items; andmeans for displaying a social interest heat map area, visuallydistinguished from and concurrently displayed with the media displayarea and the social media display area, and displaying a social interestheat map showing levels of social interest for a plurality of mediaevents from the time-based media source based on a number of socialmedia content items determined to be relevant to each of the mediaevents.
 7. The system of claim 6, wherein the social interest heat maparea displays different media event types such that the different mediaevent types each are visually distinct.
 8. The system of claim 6 furthercomprising means for determining a confidence score indicative of aprobability that the social media content items are relevant to themedia event.
 9. The system of claim 8, wherein determining theconfidence score indicative of the probability that the social mediacontent items are relevant to the media event further comprises:extracting media event features from annotations associated with themedia event; extracting social media features from the plurality ofsocial media content items; and mapping the media event to the socialmedia content items based on a relationship between the media eventfeatures and social media features.
 10. The system of claim 9, furthercomprising annotating the media event with the annotations usingmetadata instances relevant to the media event.
 11. A system forproducing a user interface of a social interest estimation system, thesystem comprising: a computer processor; and a computer-readable storagemedium storing computer program modules configured to execute on thecomputer processor, the computer program modules comprising a userinterface engine configured to graphically display: a media display areafor displaying a media event from a time-based media source, the mediaevent having a time of occurrence; a social media display area, visuallydistinguished from and concurrently displayed with the media displayarea, for displaying a plurality of relevant social media content items,the social media content items: broadcasted by a social media source,the social media source independent from the time-based media source;and determined to be relevant to the displayed media event based on arelationship between the time of occurrence and a time of authorship ofeach of the plurality of social media content items; and a socialinterest heat map area, visually distinguished from and concurrentlydisplayed with the media display area and the social media display area,for displaying a social interest heat map showing levels of socialinterest for a plurality of media events from the time-based mediasource based on a number of social media content items determined to berelevant to each of the media events.
 12. The system of claim 11,wherein the social interest heat map area displays different media eventtypes such that the different event types each are visually distinct.13. The system of claim 11, wherein the computer program modules furthercomprise a social interest estimator configured to determine aconfidence score indicative of a probability that the social mediacontent items are relevant to the media event.
 14. The system of claim13, wherein determining the confidence score indicative of theprobability that the social media content items are relevant to themedia event further comprises: extracting media event features fromannotations associated with the media event; extracting social mediafeatures from the plurality of social media content items; and mappingthe media event to the social media content items based on arelationship between the media event features and social media features.